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A technique to measure the quality of agricultural 
i^xtenslon programs by the clientele was developed and tested. The 
f study was conducted in three major steps* Firsts unstructured 

interviews of forty-eight farmers and agribusiness persons in seven 
f Ohio counties were conducted to identify those factors they 

considered when evaluating their county's extension agricultural 
t program. The forty-four factors were identified and were categorized 
: as methods, personal qualities of the agent, and quality of extension 

- information. Second, a questionnaire was sent tc 248 farmers asking 
: them to rate the importance of each of the forty-four factors. Based 
on a factor analysis of the responses, a questicnnaite was developed 
vhich consisted of forty-three factors classified in the following 
^ four sQbscales: information, extension agents, extension methods, and 
^ educational program. During the third and final step of the study, 

the questionnaire was sent to 400 farmers and agribusiness persons in 
>; eight counties while six extension prcfessictals, using a 
- paired-comparison procedure, rated the quality cf the programs in the 
\- same counties. The results of this last step indicated that the 
^ instrument and the subsoales are reliable. It is recommended that the 
{ instrument be used by others, both as an evaluation tool and for 
i further research. (The questionnaire is attached.) (EM) 
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The purpose of this ES/USDA funded research project was to 
develop and test a technique to identify output measures as perceived by 
Extension clientele. 

Limited to clientele in agricultural programs, the technique 
involved relatively unstructured pe'rsonal interviews with a sample of 
Extension agricultural clientele in Ohio, and responses to a mailed 
questionnaire from other clientele in subsequent stages of the project. 
Also involved were panels of experts - administrators and specialists in 
Ohio agricultural Extension programs. 

Although the techniques involved were rather unorthodox, making 
data analysis more difficult than in more structured types of research, 
we feel the procedures used were successful enough to reconmend their 
use with other Extension program areas. 

The primary product of the project was a 43 item scale of items 
identified by clientele as the factors they Ube in evaluating Extension 
agricultural programs in their counties- Four subscales of the instru- 
ment were classified as (1) Information, (2) Extension Agents, 
(3) Extension Methods, and (4) Educational Program. Statistical 
analysis of the instrument and the subscales showed them to be highly 
reliable. We recommend that the instrument be used by others, both as 
an evaluation tool and for further research. 

We acknowledge the excellent contributions to the success of 
the project by two graduate research assistants, Wayne M. Keffer and 
S. Alden Hilliker, 

A more detailed report is available on request to the Ohio 
Cooperative Extension Service. 



Richard E. Young 
Leader 

Studies and Evaluation 



Clarence J. Cunningham 
Assistant Director 
Staff Development and 
Program Analysis 



EXTENSION OUTPUT MEASURES 
AS IDENTIFIED BY EXTENSION CLIENTELE 



All public agencies, including the Cooperative Extension Service, 
are faced with the growing demand to demonstrate program accountability. 
Indicators of program resource input are readily available; however,, fen 
measures of Extension output effectiveness have been developed for docu- 
mentation purposes. Very few attempts have been made to determine the 
perceptions of clientele regarding evidence of program accomplishments. 

With this in mind, the Ohio Cooperative Extension Service 
submitted a proposal for research to the Extension Service, United States 
Department of Agriculture. The purpose of ths funded study was "to 
develop and test a technique to identify output measures as perceived by 
Extension clientele". 

Objectives 

The approved proposal contained two objectives; one long range 

" to identify, as perceived by clientele, the concrete evidences 

they accept as demonstrating Extension program accomplishment". The 
other objective was more inmediate " to create and test a tech- 

nique for obtaining from Extension clientele, valid output measures of 
an Extension program". 



Procedure and Findings 



Agricultural industry was the program area of the Ohio Extension 
Service that was chosen as the focus for this study. It was felt that 
resources could most efficiently be utilized by concentrating on one 
program area, rather than all four. 

The study was conducted in three major steps. In the first step 
a specific audience was defined as the focus for the research and from 
those persons a list of items that they perceived as outputs of Extension 
was determined. In the second step those items were further tested and 
analyzed as preparation for the third step during which a questionnaire 
was field tested. 

Development of Output Items 

Traditional research methods were neither planned nor used in 
conducting the study. The research team felt a need to permit clientele 
freedom to disclose their Extension evaluation techniques without bias 
of a structured interview or questionnaire. The researchers thus planned 
to utilize an unstructured form of Interview — including open-ended 
questions and extensive use of probing questions — for data gathering. 

The interview schedule was pretested with 13 farmers in two ' 
counties. The revised schedule and procedures were then used in 
Interviewing 48 farmers and agribusiness persons in seven counties. 
Efforts were made to insure that those included in the sample were 



representative of Extension agricultural clientele in the state. 
Purpose of the interviews was to elicit comments on factors that 
respondents considered when evaluating their county's Extension agricul- 
tural program. 

Two techniques during the interviews seemed to be the most 
productive. In one the interviewer asked the respondent to rate the 
quality of the program on a scale of one to seven; once the respondent 
indicated his/her choice, the interviewer asked the respondent to relate 
some of the "things* that he/she thought about in making the choice. In 
the other technique the interviewer asked the respondent to assume that 
support for the Extension agricultural program in the county had been 
lost; what arguments would the respondent use in an effort to get support 
reinstated? 

Forty-six of the interviews were successfully tape recorded and 
subsequently transcribed. A panel of 20 experts (area supervisors, state 
agricultural specialists and area agricultural agents) reviewed the trans- 
cripts with the purpose of identifying and categorizing evaluative "output 
measures". Each interview was reviewed by two panel members. In order to 
proy'de a means of quantifying the factors identified, a factoral weight 
was placed on identified itemr,. A value of 3 was given to any phrase 
noted as a factor by both experts and placed in the same category. Such 
common interpretation was classified as a "joint agreement". When both 
experts noted the comment but placed it in different categories, the 
interpretation was labeled as a "joint identification" and a value of c 
placed in each category. If only one expert saw a comment as indicating 



a measure of Extension output, it was labeled as an "identification" 
with a value of 1 placed in the appropriate category. 

Table 1 lists the forty-four factors that received weighted 
values of 15 or more. 

TABLE 1 

WEIGHTED VALUES OF EXTENSION OUTPUT' MEASURES 
IDENTIFIED BY EXTENSION CLIENTELE 

Weighted 



Rank Measure Value 

1 Information via mailing listc or direct mail 97 

2 Newspaper and magazine articles and columns 63 

3 Personal assistance from Extension Agent 49 

4 Newsletters and calendars of events 47 

5 Extension information is current 43 

6 Extension is a good source of information and help 40 
6 Formal education and training of Agent 40 

8 Agent knows where to get information and resources 39 

9 Agent relates well to community citizens 38 

10 Experience and background of agent 37 

11 Information available quickly on request 36 
11 I , formation is practical 36 
11 Radio and television programs and spots 36 
14 Agent is helpful and gets thinys done 35 
14 A source of new information and methods 35 
16 Scope of total program 34 



Weighted 

Rank Measure Value 

17 Agent helps sort out agriculture information 32 

17 Information and assistance meets needs 32 

17 Localized meetings for citizen needs 32 

20 Special subject matter meetings 31 

21 Agent is knowledgable 30 
21 Honesty and devotion of agent 30 

23 Seminars, conferences, schools, field days 29 

24 Agent or specialist come to home or farm 28 

25 Agent response to requests for assistance 27 
25 Help from agent via telephone 27 
25 Information is educational with university base 27 
25 Program meets local expectations 27 
29 Agent works hard 26 
29 Number of meetings held 26 

31 Help clientele save money 25 

32 Programs locally oriented and adapted 23 

33 Agent's ability to apply knowledge 22 

33 Agent's character is good 22 

33 Opportunity to serve on Extension Boards and 

Committees 22 

36 Bulletins and Publications 21 

36 Personal qualities of agent 21 

38 Accurate information 20 

39 Communication with public 18 
39 Demonstrations and testing programs 18 
39 Agent respected by farmers 18 
39 Programs for special clientele groups 18 
39 Staff involvement in civic activities 18 
44 Improved standard of living 15 
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Observation of these items shows that interviewees tended to use 
three categories of factors as they evaluated their county Extension 
agriculture programs — methods, personal qualities of the agent, and 
quality of Extension information. 

Testing and Categorizing Items 

The second phase of the study had as its major product a question 
naire of valid output items placed in categories that would help both 
respondents and those who would analyze the data. 

This phase was carried out by asking a sample of 248 farmers 
(none of whom were involved in the earlier interviews) to rate how 
important each of the 44 items from the first phase of the study was as 
a measure of a county's agricultural program. Nearly 75% of the sample 
returned the mailed questionnaire. 

Begause one purpose of this phase was to categorize the items, 
the statements were in random order on the instrument to which the 
farmers responded. Factor analysis was used as a helpful aid to this 
phase; it helped measure the reliability of the instrument and its 
sub-scales (categories) and in decisions to keep or delete individual 
items. 

Mean importance scores were calculated for each item on the 
basis of a five-point zcale from extremely important (5) to not impor- 
tant (1). The range was from 2.76 to 4.74 with only two items below 
3.00. No item was deleted on the criterion of lack of importance. 



After several factor analysis runs and visual inspection of their 
results, the instrument consisted of 43 items ("Agent is helpful and gets 
things done" was the deleted item) in four categories • 

Reliability tests showed a Kuder-Richardson Test Reliability 
Score of .93 for the total instrument. Scores for the sub-scales ranged 
from .70 for the eight-item Program sub-scale to ,85 for the 14-item 
Information one* 

Field Testing the Instrument 

The third, and final, step in the project was to test the 
instrument that resulted frOm the preced1ng|;§teps with a sample of 
farmers and agribusiness people in a small number of Ohio counties. An 
attempt was also made to validate the instnjment by comparing clientele 
evaluations with evaluations of county agricultural programs by knowl- 
edgeable Extension professionals 

Four hundred farmers and agribusiness persons (50 from each of 
the eight counties in one Extension area) were mailed a copy of the 
instruiient and asked to respond • Forty-one percent did so within three 
weeks. A panel of six Extension professionals (the area supervisor, 
area agents who worked in the area and the associate state agriculture 
leader) rated the quality of the agricultural programs in the eight 
counties using a paired- comparison procedure. 

The professional panel agreed on their rankings of the eight 
counties (Kendall *s W tested the agreement). And although thers was 
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a tendency toward agreement between the ranking of the professionals 
and the ranking of counties according to clientele responses, the 
Spearman rank correlation coefficient showed the agreement not to be 
great enough to be statistically significant. (Required level of 
significance was .05.) 

On a scale of 5 to 1 — from excellent to poor — the range In 
mean item scores was from 3,54 (good +) to 4.43 (very good +). Mean 
scores were calculated for each of the four sections and for the total 
instrument. They were: 

Information - 3.95 

Agent - 4.12 

Methods - 3.75 

Program - 3.76 

Total - 3.91 

To check on internal consistency, Kuder-Richardson test 
reliability scores were calculated for each of the four sections and for 
the total instrument. Table 2 shows that each score was very high, even 
for the eight-item Program scale. The entire instrument was extremely 
reliable; each sub-scale also could stand alone as a highly reliable 
instrument. 
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TABLE 2 

SUMMARY OF INSTRUMENT INTERNAL RELIABILITY 
N = 162 

Section No. of Items ' ' K-R Test Reliability 



Information 14 .96 

Agent 11 .96 

Methods 10 .94 

Program 8 .94 

Total 43 .98 



Tabls 3 demonstrates the high correlation between the total 
instrument and each of the sub-scales. This shows that each sub-scale 
tended to measure the same thing as the total instrument did. All other 
inter-relationships are also shown, and while the correlations are not as 
strong as those with the total scale, they do show substantial consistency 
among the various parts of the instrument. 

TABLE 3 

CORRELATIONS AMONG SUB-SCALE AND 
TOTAL INSTRUMENT SCORES 
N = 162 





Information 


Agent 


Methods 


Program 


Total 


Information 


1.00 


.86 


.81 


.86 


.95 


Agnnt 




1.00 


.80 


.80 


.93 


Methods 






1.00 


.88 


.92 


Program 








1.00 


.93 


Total 










1.00 
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From the data in Tables 2 and 3, it Is apparent that, for the 
evaluator who might be interested in a short, reliable instrument that 
would give a measure of the quality of a county's Extension agricultural 
program, any one of the four sub-scales should do quite well, though the 
Information sub-scale would be best. On the other hand, if data are 
needed to measure the quality of other factors in the program — factors 
that this project showed to be important as seen by clientele — then 
all four sub-scales would need to be used. 

Conclusions 

Each of the two objectives of the project was attained, but 
neither to the extent, nor in quite the way anticipated. 

The instrument that resulted from the project consisted of 43 
"things" that Extension agricultural clientele in Ohio said they 
considered as they evaluated the quality of Extension agricultural pro- 
grams in their couricies. The extent to which those items could be 
termed "concrete evidences" was of concern to the researchers who 
anticipated that clientele would identify indicators that might be 
measured without the necessity of having clientele respond to an 
instrument. 

A good share of other studies in which clientele have rated the 
effectiveness of Extension programs have asked clientele to rate the 
extent to which state or national objectives were being met. This 
project showed that Ohio agricultiral clientele used criteria other 
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than state or national objectives in their evaluations. They were more 
concerned with the quality of the information they received, the person- 
ality and character of their county agricultural agent, and the quality 
and quantity of methods they saw being used by the agent. 

The project was limited to agricultural clientele in Ohio and 
thus generalizations beyond that state would be risKy- On the other hand, 
the researchers felt confident that similar responses to their procedures, 
would be found in other states. 

The second project objective contained two components 

1) developing and testing a technique which would result in 2) valid out- 
put measures. 

The technique and its procedures involved unorthodox methods. 
And on reflection they were successful. 

The relatively unstructured interviews, though difficult to 
conduct — and even more difficult to analyze — did permit clientele to 
relate, as well as they could on short notice, the factors they considered 
when evaluating their county agricultural programs. 

All interviews were tape recorded; no resistance to that procedure 
was encountered. The recorded interviews were transcribed to facilitate 
their analysis; this was essential to the procedure. The panel of experts 
who read the transcribed interviews and identified "output measures" 
mentioned by interviewees found their task to be time-consuming and some- 
what tedious; yet the researchers felt this to be a necessary step in the 
procedure. 

n 



The verification by clientele of the identified items as "important" 
indicators of Extension output was another essential part of the process. 
Respondents were able to do this, but they had difficulty responding from 
a frame of reference more general than their own county. Factor analysis 
and item analysis of the i^esponses helped place it&ns in logical categories 
and to confirm that the instrument was high in internal consistency. 

The final step in the project — field testing of the instrument — 
yielded evidence to show the instrument to be extremely reliable. It 
seemed to be able to measure county-by-county differences in program 
quality consistently. In addition, a computer generated analysis of each 
county's data was provided for the county agent. 

Though the procedure was somewhat unorthodox, and though it was 
time consuming, the researchers felt that each step was essential and 
that the final result was worth the effort expended. 

The extent to which the items in the final instrument are "valid 
output measures" of Ohio county Extension agricultural programs may need 
further testing. But they did result from personal interviews with 
clientele during which every effort was made to elicit measures of 
Extension output. And all subsequent analysis and testing served to 
enhance rather than diminish the validity and reliability of those items. 
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Recommendations 



The primary recommendation from the project is that the instru* 
ment be used as a device for measuring the quality of county agricultural 
Extension programs. 

Related to the above recommendation is this: when the instrument 
is used, further analysis should be made of the resulting data as a means 
to add further confidence in it. A question might be added to ask respon- 
dents for a rating of their overall evaluation of their county Extension 
agricultural program; the correlations between individual item scores and 
category scores to this general rating would provide additional evidence 
of reliability. 

The instrument could be used as a component in other Extension 
related research. For example, what relationships would be shown between 
respondent scores on this instrument and such clientele variables as 
extent of participation in Extension programs, participation on Extension 
committees, farm income, age, and farm enterprise? 

If used in several counties, relationships between county mean 
scores on the instrument (and its subscales) and input variables such as 
time spent by Extension professionals on agricultural programs, bulletins 
distributed, radio programs presented, and personal data regarding the 
agent might be explored. Other county data, such as average crop yields 
and agricultural income could also be compared with instrument scores. 

In those states where the use of key result indicators has been 
tried, the resulting measures for each county might be compared with 
clientele evaluations of their county Extension agricultural program as 
measured by this instrument. 
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Clientele evaluations of the extent to which county, state and/or 
national objectives are being attained might be compared with their scores 
on the instrument. 

A final reconmendation is that the procedures used in this project 
be tried with the other program areas in Extension — home economics, 
community resource development and 4-H youth. 



INSTRUMENT 
FOLLOWS 
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For each of the following items, please circle the response in the column 
which best expresses your evaluation regarding that element of the tot^l E)ctension 
. agricultural program in your county. 





Item 


1 Excellent 


Very 
Good 


Good 


Pair 


Poor 


A. INFORMATTON 












1. 


Help trem agent by telephone. 




VO 


G 


F 


P 


2. 


Agent knovs where to get 
information and resources. 


E 


VO 


0 


P 


P 


3. 


Information is accurate. 


E 


VO 


0 


P 


P 


k. 


Quality of bulletins and 
publications. j 


E 


VO 


0 


P 


P 


5. 


Information and assistance meets 
individual needs. 


E 


VO 


0 


P 


P 


6. 


Agent is knowledgeable. 


E 


VO 


0 


P 


P 


7. 


Information is current. 


E 


VO 


0 


P 


P 


8. 


assistance. 


E 


VO 


0 


P 


P 




xcxowfiiAX Q>ooXo vCbIlL>c XxOiU juKoensxon 

agent. 


E 


VO 


0 


P 


P 


10 • 


Extension as & soiirce of new 
information and methods. 


E 


VO 


0 


P 


P 


11 • 


Inforaation is practical. 


E 


VO 


0 


P 


P 


12. 


Information is quickly available 
on request. 


E 


VG 


G 


F 


P 


13- 


Extension is a good source of 
quality information and help. 


E 


VG 


G 


F 


P 


Ik. 


Agent helps sort out conflicting 
agricultural information. 


E 


VG 


G 


F 


P 
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Item 


Excellent 


Very 
Good 


Good 


Pair 


Poor 


B. EXTENSION AGENOS 












15. Personal qualities of agent. 


E 


VG 


G 


F 


P 


16. Hov agent relates to local 
citizens . 


E 


VG 


G 


F 


P 


17. Agent's ability to apply 
knoif ledge. 


E 


vo 


G 


F 


P 


18. Iftird working agent. 


E 


VG 


G 


F 


P 


19. Formal education and training of 
agent. 


E 


VG 


G 


F 


P 


20. Experience and backgrotind of 
agent. 


E 


VG 


G 


F 


P 


21. Honesty and devotion of agent. 


E 


VG 


G 


F 


P 


22. Respect for agent by local 
farmers . 


E 


VG 


G 


F 


P 


23. Involvement of agent in civic 
affairs. 






\i 


r 


x> 

r 


2k. Character of agent. 


E 


VG 


G 


F 


p 


25. Agent's comniunication with the 
public. 


E 


VG 


G 


F 


p 


C. QCTEICION METHODS 












26. Narspaper and magazine articles 
and coliUDns. 


E 


VG 


G 


F 


p 


27. Radio and television programs 
and spots. 


E 


VG 


G 


F 


p 


29. Information received via mailing 
lists or direct mall. 


E 


VG 


G 


F 


p 


29. Seminars, conferences, schools, 
and field days. 


E 


VG 


G 


F 


p 


30. Programs for special clientele 
groups . 


E 


VG 


G 


F 


F 



Item 1 


Excellent 


Very 
Good 


Good' 


Pair 


Poor 


31. 


Nuxtiber of inset jngs held. 


E 


VG 


G 


F 


p 


32. 


Visits by agents or specialists 
to farm. 


E 


VG 


G 


F 


p 


33. 


Newsletters and calendars of 
eveats . 


E 


VG 


G 


F 


p 


3H. 


Demonstrations and testing 
programs . 


E 


VG 


G 


P 


p 


35. 


Special subject natter meetings. 


E 


VG 


G 


F 


p 


D. EDUCATIONAL PROGRAM 












36. 


Program meets local expectations. 


E 


VG 


G 


F 


p 


37. 


Information is educational with a 
viniversity base. 


E 


VG 


G 


F 


p 


38. 


Frogrflffi is adapted and oriented 
locally. 


E 


VG 


G 


F 


p 


39. 


Meetings that meet needs of local 
citizens . 


E 


VG 


G 


F 


p 


ko. 


Citizens' opportunities to serve 
on Extension boards and 
committers . 


E 


VG 


G 


F 


p 


hi. 


Scope of the total Extension 
agricultural program. 


E 


VG 


G 


F 


p 


l42. 


Extension's influence in an 
improved standard of living for 
local citizens. 


B 


VG 


G 


F 


p 


U3. 


Extension's efforts to help 
clientele save money. 


E 


VG 


G 


F 


p 



Thank you for your time and fkitnk responses to the above statements. Please 
check again to see that you have responded to each Item. After you have done so^ 
place the questionnaire In the provided envelope. Please nail both the envelope 
and the coded card as soon as possible to: 
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